A robot can be programmed to perform actions with respect to various types of objects. For example, a robot can be programmed to open a door or pick up and move an object. To perform such tasks, an autonomous or semi-autonomous robot should detect the object and determine how to perform the task on or using the object.